Statistical Learning Theory Lecture 22 ( Nov 7 , 2013 ) Stochastic Multi - Armed Bandits

In this lecture we will start to look at the multi-armed bandit (MAB) problem, which can be viewed as a form of online learning in which the learner receives only partial information at the end of each trial. Specifically, in an n-armed bandit problem, there are n different options that can be chosen, or n different arms that can be pulled, on each trial; each of these arms, when pulled, generates some reward (or loss). On each trial, the learner selects one arm to pull, and observes the reward associated with only the pulled arm; the rewards associated with the other arms remain hidden from the learner (it is in this sense that the learner receives only partial information). The goal of the learner is accumulate as much reward as possible over a sequence of trials, e.g. compared to the best fixed arm in hindsight. Such problems arise in a variety of applications, e.g. in clinical trials, where a doctor must select one of n available treatments to give to each incoming patient, and gets to observe the outcome of only the particular treatment delivered to the patient; in ad placements, where one needs to decide which of n available ads to display to any given user, and gets to observe the click behavior/revenue generated only for the ad displayed; and so on. In all these applications, there is an inherent trade-off between exploration (trying a new arm that might yield better rewards) and exploitation (selecting an arm that has been observed to give good rewards so far), and any good MAB algorithm must somehow balance these two aspects.